Lifetime-based memory management for distributed data processing systems
نویسندگان
چکیده
منابع مشابه
Lifetime-Based Memory Management for Distributed Data Processing Systems
In-memory caching of intermediate data and eager combining of data in shuffle buffers have been shown to be very effective in minimizing the re-computation and I/O cost in distributed data processing systems like Spark and Flink. However, it has also been widely reported that these techniques would create a large amount of long-living data objects in the heap, which may quickly saturate the gar...
متن کاملEntropy-based Consensus for Distributed Data Clustering
The increasingly larger scale of available data and the more restrictive concerns on their privacy are some of the challenging aspects of data mining today. In this paper, Entropy-based Consensus on Cluster Centers (EC3) is introduced for clustering in distributed systems with a consideration for confidentiality of data; i.e. it is the negotiations among local cluster centers that are used in t...
متن کاملPolicy-based Distributed Data Management Systems
The integrated Rule-Oriented Data System (iRODS) is a data grid that manages distributed data. The iRODS system is used to support data sharing, data publication, data preservation, and data analysis. These data management applications depend upon very similar data manipulation capabilities, but differ in terms of their management policies. The iRODS data grid organizes distributed data into a ...
متن کاملROLP: Runtime Object Lifetime Profiling for Big Data Memory Management
Low latency services such as credit-card fraud detection and website targeted advertisement rely on Big Data platforms (e.g., Lucene, Graphchi, Cassandra) which run on top of memory managed runtimes, such as the JVM. These platforms, however, suffer from unpredictable and unacceptably high pause times due to inadequate memory management decisions (e.g., allocating objects with very different li...
متن کاملVStorm: Video Traffic Management By Distributed Data Stream Processing Systems
Recent published work shows that Quality of Experience (QoE) has become one of the major concerns in the area of large-scale multimedia Internet services. Due to the significant increase of video traffic and continuously growing need of video quality experience, we need new management platforms for multimedia traffic. Recent studies propose a variety of solutions that address this challenge. Ho...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Proceedings of the VLDB Endowment
سال: 2016
ISSN: 2150-8097
DOI: 10.14778/2994509.2994513